Feat: Pre-quantized LLM model support #3740
Conversation
keehyuna
changed the title
keehyuna
requested review from
narendasan and
peri044
and removed request for
narendasan
keehyuna
force-pushed
the
quant_llm_merged
branch
from
62cb0f2 to
1f1cf7f
Compare
| return model | ||
|
|
||
|
|
||
| class TensorRTQuantizedLinear(torch.nn.Module): |
There was a problem hiding this comment.
@peri044 Is this something we might want to upstream to ModelOpt in the future?
There was a problem hiding this comment.
Or pull into main torch-tensorrt as a pass?
There was a problem hiding this comment.
I guess its somewhat HF specific, so remaining in this tool would make sense but are there some parts we could make generic for any sort of quantization workflow (e.g. torchao)?
There was a problem hiding this comment.
Thanks. I think quantize_model() can be moved to function like torch_tensorrt.dynamo.quantize(). Currently investigating how to separate the calibration data path from the quantization logic
tools/llm/quantize_utils.py
Outdated
|
|
||
| hf_quant_algo = hf_quant_config.pop("quant_algo", None) | ||
| if hf_quant_algo != "FP8" and hf_quant_algo != "NVFP4": | ||
| raise RuntimeError("Only FP8 or NVFP4 quantization is supported") |
There was a problem hiding this comment.
How would it be different for MXFP4?
There was a problem hiding this comment.
looked at quantization cfg in modelopt
NVFP4_DEFAULT_CFG NVFP4 has E4M3 scales and a block size is 16.
MXFP4_DEFAULT_CFG MXFP4 has E8M0 scales and a block size is 32.
keehyuna
force-pushed
the
quant_llm_merged
branch
from
19a4070 to
303775e
Compare
github-actions
bot
added
component: api [Python]
Description
Support pre-quantized HF models and post-training quantization (PTQ) option for run_llm.py
Fixes # (issue)
Type of change
Checklist: